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ABSTRACT 
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building  the  Army  budget.  The  Value  Added  Analysis  concept  uses  a  family  of  models  to 
estimate  an  alternative  system's  contribution  to  the  Army's  effectiveness  using  a  multi-attribute 
value  hierarchy.  A  mathematical  optimization  model  is  then  used  to  simultaneously  determine  an 
alternative's  cost-benefit  and  to  identify  an  optimal  mix  of  weapon  systems  for  inclusion  in  the 
Army  budget. 
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1.  INTRODUCTION 

Each  year  the  US  Army  considers  numerous  proposed  programs  to  procure  new  weapon 
systems  and  other  equipment.  These  proposals  all  have  proponents  who  assert  that  their 
systems  satisfy  Army  needs  and  improve  significantly  upon  predecessor  systems.  These 
assertions  are  backed  with  analyses  conducted  by  the  command  that  is  responsible  for  the 
specific  mission  areas  that  are  to  be  improved,  and  the  total  cost  of  all  proposals  would  be 
billions  of  dollars  annually.  However,  Army  procurement  budgets  have  been  reduced  by  half 
since  the  1980’s  and  are  not  sufficient  to  fund  all  the  systems  and  proposals.  Funding  pressures 
will  continue  to  intensify  as  the  perception  of  threats  around  the  world  changes  and  the  federal 
budget  deficit  continues  to  loom  over  all  public  spending.  As  a  result,  solid  and  impartial 
analysis  is  needed  at  the  Department  of  the  Army  level  to  provide  insightful  recommendations  on 
which  systems  to  fund. 

This  paper  describes  a  modeling  methodology  called  Value  Added  Analysis  (VAA),  which 
helps  the  US  Army  meet  this  critical  analysis  need.  The  Value  Added  means  incremental  return 
on  a  system’s  investment  based  on  its  contribution  to  the  Army’s  effectiveness  in  performing  its 
missions.  VAA  uses  a  family  of  simulation,  statistical,  and  decision  analytic  models  to  estimate 
the  value  added  and  cost  of  alternative  procurement  programs.  Optimization  techniques  are  then 
applied  to  identify  a  theoretically  optimal  mix  of  weapon  systems  and  equipment.  The 
"optimal"  solution  is  then  used  as  a  foundation  for  parametric  analyses  that  provide  senior 
leaders  insight  into  the  implications  of  system  trade-offs  and  changes  in  planning  guidance. 

Prior  to  VAA,  the  process  that  evolved  over  many  years  of  procurement  budgeting  was  to 
convene  a  separate  panel  to  evaluate  and  prioritize  alternative  weapon  systems  and  other 
equipment  for  each  functional  area  of  the  Army  (fire  support,  maneuver,  aviation,  etc.).  These 
panels  acted  independently,  and  the  process  did  not  include  adequate  means  of  integrating  the 
separate  panels’  recommendations  into  a  balanced  Army  program.  VAA  overcomes  this 
shortcoming  of  the  traditional  approach  by  evaluating  the  value  added  of  all  the  alternative 
systems  in  a  consistent  and  objective  manner,  allowing  optimization  of  the  procurement  program. 

Every  other  year,  the  Army  produces  its  funding  plan  called  the  Program  Objective 
Memorandum  (POM).  VAA  has  been  used  since  1991  to  support  the  decisions  made  by  the 
Office  of  the  Deputy  Chief  of  Staff  for  Operations  and  Plans  and  the  Programs  Analysis  and 
Evaluation  Directorate,  which  have  the  primary  staff  responsibility  for  preparing  the  POM  and 
making  recommendations  to  senior  leaders. 

The  remainder  of  the  paper  describes  VAA’s  overall  framework  and  the  evolution  of  the 
process  over  the  past  half  decade. 
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2.  OVERVIEW  OF  THE  VAA  ANALYTICAL  FRAMEWORK 


The  VAA  analytical  framework  provides  a  road  map  for  conducting  Army  procurement 
program  tradeoff  and  capital  budgeting  analyses.  It's  structure  is  similar  in  nature  to  other  efforts 
of  this  type.  For  example,  Hall,  et.  al.  [7]  describe  a  similar  modeling  framework  that  was  used  to 
assist  in  program  funding  decision  making  at  the  National  Institutes  of  Health.  Peereboom,  et.  al. 
[18]  used  related  techniques  to  select  a  portfolio  of  environmental  programs  for  the  U.S. 
Department  of  Energy.  Another  approach  for  the  solution  of  the  Army  capital  budgeting 
problem  for  procurement  programs  has  been  suggested  by  Anderson  [1].  His  methodology 
departs  from  the  one  described  here  in  that  it  considers  a  broader  variety  of  procurement 
programs,  but  it  does  so  in  a  less  detailed  fashion. 

Figure  1  depicts  the  six  modules  in  the  VAA  analytical  framework.  This  framework  could 
be  used  for  any  of  the  appropriations  (operations  and  maintenance;  military  construction; 
research,  development,  and  acquisition  (RDA);  etc.).  The  scope  of  the  study  effort  has 
historically  been  limited  to  selected  major  weapon  systems.  These  are  the  weapon  systems  that 
are  important  in  the  Army  Modernization  Plan  and  funded  using  RDA  appropriations.  This 
scope  captures  over  fifty  percent  of  the  dollar  value  of  the  RDA  appropriation,  addresses  the 
"big  issues",  and  focuses  analyses  into  key  decision  areas. 


Feedback 


Figure  1.  Value  Added  Analysis  Methodology  Overview 
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The  VAA  methodology's  six  modules  correspond  to  a  loosely  coupled  process  consisting 
of  the  following  steps:  (1)  Definition  of  the  context  of  the  problem  and  identification  of 
important  issues  to  be  studied;  (2)  Determination  of  the  benefit  of  procuring  the  systems  based 
on  their  contribution  to  the  combat  effectiveness  of  the  force,  the  deployability  of  the  force,  and 
the  sustainability  of  the  force;  (3)  Estimation  of  the  of  the  life  cycle  costs  for  each  alternative 
program;  (4)  Analysis  of  the  cost-benefit  of  the  systems  by  means  of  a  mathematical  program; 

(5)  Evaluation  of  the  recommended  system  mix  in  various  theaters  of  operation;  and  (6) 
Presentation  of  the  results  to  the  decision  makers. 

Each  of  the  steps  in  the  VAA  framework  is  essentially  a  separate  analysis,  the  output  of 
which  provides  input  to  a  subsequent  module  and  analytic  insights  that  are  of  value  to  individual 
elements  of  the  Army  Staff  Each  of  the  modules  will  be  addressed  in  the  sections  that  follow. 

3.  ISSUE  DEFINITION 

The  Issue  Definition  Module  has  two  purposes.  The  first  is  to  establish  a  broad  context 
for  the  study  in  terms  of  time  frames  and  scenarios  to  be  analyzed,  the  list  of  major  item  systems 
to  be  considered,  the  criteria  by  which  system  contribution  should  be  judged,  and  the  measures  of 
effectiveness  (MOE)  to  be  used.  These  factors  define  how  the  benefit  of  the  candidate  systems  is 
to  be  articulated  numerically.  As  such,  issue  definition  will  have  tremendous,  often  subtle  impact 
on  the  perceived  value  of  a  potential  weapon  system.  Therefore,  it  is  essential  that  decision 
makers  understand  the  implication  and  importance  of  this  stage  in  the  process.  The  second 
purpose  involves  identifying  the  very  specific  “What-If  ’  questions  that  arise  in  the  POM 
building  process.  Both  of  these  aspects  are  discussed  in  this  section. 

3.1  Study  Context 

The  VAA  study  considers  a  15  year  time  horizon  corresponding  to  the  years  considered  in 
Army  acquisition  planning.  This  time  horizon  is  considered  in  two  parts.  The  first  five  years  are 
called  the  POM  years,  and  the  following  ten  years  are  known  as  the  extended  planning  period 
(EPP).  Over  that  time,  new  systems  will  be  introduced  into  the  force  and  changes  will  occur  in 
the  enemy  threat.  As  a  result,  the  expected  contribution  of  the  systems  under  consideration  by 
the  VAA  process  will  fluctuate.  Ideally  we  would  like  to  predict  the  performance  of  the  force 
and  the  relative  contribution  candidate  systems  in  every  year  of  the  time  horizon,  in  every 
conceivable  scenario.  Even  if  it  were  possible  to  articulate  all  possible  scenarios,  the  cost  in  time 
and  dollars  to  model  every  scenario  explicitly  in  this  manner  would  be  prohibitive. 

A  natural  way  of  choosing  the  years  to  perform  the  evaluation  is  based  on  the  two  parts  of 
the  planning  horizon.  Assuming  a  two  year  lag  between  procurement  and  fielding  of  new 
systems,  all  systems  procured  in  the  POM  period  would  be  available  on  the  battlefield  by  the 
seventh  year  of  the  time  horizon.  Similarly,  all  the  systems  procured  through  the  EPP  would  be 
available  for  combat  two  years  after  the  time  horizon.  Therefore,  the  evaluation  is  performed  and 
the  candidate  systems  are  simulated  explicitly  using  an  appropriate  enemy  threat  force  for  two 
years  after  the  POM  period  and  two  years  after  the  EPP.  The  values  of  relevant  MOE  for 
intermediate  years  are  estimated  using  linear  interpolation. 
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The  Defense  Planning  Guidance,  issued  by  the  Secretary  of  Defense,  identifies  scenarios 
that  provide  the  foundation  for  the  Army's  modeling  and  analysis.  In  VAA  two  combat  oriented 
scenarios  are  currently  modeled,  each  in  two  time  frames.  Future  iterations  of  Value  Added 
Analysis  could  consider  other,  more  frequently  encountered  situations  such  as  low  intensity 
conflict  and  humanitarian  relief  missions.  Nothing  inherent  in  the  methodology  precludes  these 
scenarios  from  being  included,  given  that  sufficient  resources  and  time  are  available  to  estimate 
the  expected  contribution  of  the  candidate  programs  in  these  situations. 

The  specific  list  of  procurement  programs  to  be  considered  has  been  historically  based  on 
the  following  considerations.  First,  the  study  sponsors  were  primarily  interested  in  high  cost 
combat  systems.  These  systems  spanned  the  entire  range  of  combat  functions  and  included  new 
and  upgraded  combat  vehicles,  artillery,  helicopters,  intelligence  systems,  logistics  systems,  and 
command  and  control  systems.  Second,  some  procurement  decision  was  likely  to  be  needed 
regarding  each  program  in  the  analysis.  If  it  was  generally  agreed  that  the  decision  to  fund  or  not 
fund  a  particular  program  had  already  been  made,  the  program  was  not  included  on  the  list. 
Programs  with  already  approved  funding  were  included  in  the  base  case,  and  were  modeled  in 
every  simulation  run.  Finally,  a  credible  job  of  modeling  each  system  was  required.  Some  of  the 
new  concepts  in  command  and  control,  for  example,  were  omitted  from  the  list  due  to  our 
inability  to  model  them.  Ultimately,  a  list  of  about  50  programs  were  selected  to  be  analyzed. 

The  selection  of  the  specific  criteria  and  measures  of  effectiveness  used  to  quantify  a 
system's  level  of  performance  will  be  discussed  in  subsequent  sections. 

3.2  Specific  Issues  for  Analysis 

The  second  part  of  the  issue  definition  module  involves  detailed  questions.  These  might  be 
specific  questions  regarding  individual  weapon  systems,  groups  of  systems,  perturbations  to 
available  funding,  or  changes  in  other  model  parameters.  The  intent  of  these  questions  is  to 
develop  insight  into  the  tradeoffs  between  different  mixes  of  the  systems  that  might  be  procured 
or  to  assess  the  impact  of  changes  in  planning  constraints.  Normally,  these  issues  are  identified 
as  excursions  to  the  primary  analysis  effort. 

Historically,  the  questions  were  specific  and  based  on  results  of  previously  presented 
analyses.  Examples  of  these  “What-If  ’  questions  are:  If  this  maneuver  system  that  was  not 
recommended  for  procurement  is  included  in  the  program,  what  programs  should  then  not  be 
funded?;  or.  How  would  the  recommendation  be  changed  if  the  available  funds  were  reduced  by 
$500  million  per  year  in  1998-2000? 

Our  experience  has  also  been  that  almost  every  excursion  produces  a  series  of  subsequent 
questions  which  establish  a  feedback  loop  in  the  analysis.  This  interactive  process  was  normally 
accomplished  within  24-48  hours  to  stay  inside  of  the  decision  cycle  that  accompanies  critical 
stages  of  the  planning  process. 


4.  COMBAT  EFFECTIVENESS 

The  Army  invests  in  new  weapon  systems  and  logistical  support  systems  to  improve  the 
effectiveness  of  our  forces  in  combat.  All  of  the  proposed  programs  are  either  improvements 
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over  an  existing  system  or  they  are  new  systems  that  are  designed  to  meet  a  validated 
requirement  on  the  battlefield.  The  requirements  represent  a  heterogeneous  set  of  contributions, 
many  of  which  contribute  indirectly  to  the  battle.  VAA  uses  experiments  conducted  using 
combat  simulation  modeling  to  estimate  homogeneously  the  relative  contribution  of  proposed 
systems.  The  following  provides  a  description  of  this  estimation  process. 

4.1  Combat  Simulation  Modeling 

For  a  combat  simulation  output  to  be  useful,  the  model  used  must  represent  the  candidate 
systems  at  an  appropriate  level  of  resolution.  For  this  application  of  Value  Added  Analysis,  a 
corps-level  combat  model  was  used.  The  use  of  this  level  of  resolution  was  important  for  several 
reasons.  First,  US  Army  planning  typically  calls  for  the  deployment  of  corps  or  corps-like 
structures  to  the  areas  of  the  world  and  the  potential  conflicts  being  used  as  the  basis  for  the 
combat  modeling.  Second,  a  corps-sized  force  makes  use  of  a  fuller  mixture  of  maneuver,  fire 
support,  combat  environment,  logistics,  and  command  and  control  (C2)  than  lower  (e.g.,  division) 
level  organizations.  Finally,  corps-level  simulations  have  the  sensitivity  necessary  to  evaluate 
weapon  systems  and  their  effect  in  a  level  of  detail  that  would  be  missing  in  a  more  aggregated 
model.  The  Corps  Battle  Analyzer  (CORBAN)  [24]  was  chosen  to  simulate  the  overall  battle. 

CORBAN  is  a  time-stepped,  closed-loop,  deterministic  simulation  that  models  combat  at 
the  corps/army  level.  It  can  simulate  battles  ranging  in  size  from  one  battalion  versus  another,  to  a 
battle  of  eight  US  corps  opposing  eight  "Red"  armies.  Resolution  of  the  model  is  at  the  battalion 
level.  However,  individual  weapon  systems  and  their  characteristics  are  modeled.  Maneuver  and 
fire  support  systems  can  be  modeled  directly  in  CORBAN.  However,  the  air  defense  and 
logistics  systems  required  the  use  of  functional  area  models  and  other  off-line  analysis  to  provide 
calibration  input  to  CORBAN  so  that  the  effects  of  these  systems  could  be  captured. 

A  base  case  is  established  for  each  scenario/time  frame  using  forces  equipped  with 
systems  that  are  known  to  be  available  during  the  particular  year  of  interest.  Excursions  are  then 
developed  by  replacing  base  case  systems  with  new  weapon  systems,  or  adding  new  systems,  to 
produce  new  results  that  are  compared  with  the  base  case  results.  The  results  of  these  excursions 
are  used  to  determine  the  contribution  of  each  candidate  system  to  the  effectiveness  of  the  force. 


4.2  Combat  Measures  of  Effectiveness 

The  choice  of  combat  measures  of  effectiveness  (MOE)  is  always  a  difficult  one.  An 
advantage  of  the  VAA  methodology  is  that  it  is  not  limited  to  only  one  MOE.  Therefore,  a  set  of 
MOE  can  be  selected  that  are  deemed  important  by  the  study  sponsors.  Ideally  that  set  of  MOE 
will  characterize  the  performance  of  the  force  as  a  whole  and  be  sensitive  to  the  contributions  of 
the  candidate  systems  under  consideration. 

During  prototype  implementation  of  the  VAA  methodology,  data  on  several  MOE  were 
collected.  These  included  loss  exchange  ratio  (LER),  which  is  the  ratio  of  red  major  weapon 
systems  lost  to  that  of  blue;  correlation  of  forces  and  means  (COFM),  a  measure  of  red  force 
strength  in  an  attack  corridor  with  respect  to  the  blue  combat  strength  in  the  same  corridor;  blue 
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force  surviving  (BPS)  and  fractional  exchange  ratio  (PER),  which  are  defined  below;  and  red  force 
movement  (RPM),  a  measure  of  the  distance  moved  by  the  red  force  computed  using  its  center  of 
mass.  Of  these,  all  but  BPS  were  significantly  correlated,  indicating  that  some  relation  exists 
between  them.  Since  we  would  like  as  much  independence  between  the  MOE  as  possible,  we 
chose  only  one  measure  among  the  correlated  MOE  to  include  in  the  analysis. 

Practional  exchange  ratio  was  chosen  as  a  method  for  determining  the  lethality  of  the  force, 
measuring  the  proportion  of  red  losses  compared  to  the  proportion  of  blue  losses.  PER  is  a 
commonly  used  measure  in  the  analysis  of  weapon  systems  and  is  familiar  to  the  receivers  of  the 
analysis.  Also,  using  historical  combat  data,  Helmbold  [8]  showed  that  PER  is  a  good  indicator 
of  success  in  battle. 

PER  is  computed  as  follows.  Let  Ib  be  the  index  set  of  blue  systems  included  in  the 
calculation.  This  set  of  systems  is  limited  to  combat  vehicles,  self-propelled  artillery,  and 
aviation  systems.  These  systems  account  for  most  of  the  combat  power  of  the  force.  Let  Ir  be 
the  index  set  of  the  corresponding  red  systems.  With  'Pf  being  PER  we  have 

i€lB  ielg 


where  bj  =  quantity  of  blue  system  i  lost  during  the  simulated  battle, 

Bi  =  quantity  of  blue  system  i  available  at  the  start  of  the  simulation, 
n  =  quantity  of  red  system  i  lost  during  the  simulated  battle,  and 
Ri  =  quantity  of  red  system  i  available  at  the  start  of  the  simulation. 

Despite  its  stated  advantages,  the  mathematical  form  of  PER,  namely  a  ratio  of  ratios,  is  a 
potential  source  of  ineonsistency.  Care  must  be  taken  to  control  unwanted  variation  between 
simulation  runs.  A  great  deal  of  detailed  output  analysis  was  required  to  ensure  consistency 
between  the  runs.  In  particular,  the  establishment  of  a  suitable  stopping  criterion  was  found  to 
be  important  in  controlling  irrelevant  variation. 

Where  PER  is  a  measure  of  success  in  battle,  BPS  is  a  measure  of  the  combat  power 
remaining  after  the  battle,  available  for  a  subsequent  conflict.  As  such,  BPS  represents  a  measure 
of  the  expenditure  of  resources  needed  to  accomplish  some  mission. 

BPS  is  computed  as  follows.  Using  the  above  notation,  and  with  'Fb  being  BPS  we  have 

•  _T 


where  I  =  set  of  all  blue  systems,  and  =  combat  capability  weight  of  system  i. 

Note  that  the  parameters  are  data  used  in  the  CORBAN  decision  process.  These 
parameters  are  used  to  reflect  the  fact  that  different  items  of  equipment  used  by  combat  units 
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have  different  levels  of  importance.  A  great  deal  of  controversy  exists  in  the  combat  modeling 
community  regarding  the  use  of  these  static  measures,  and  ideally  we  would  avoid  their  use. 
Nevertheless,  some  accommodation  must  be  made  to  the  consideration  of  the  relative  importance 
of  the  various  pieces  of  equipment  used  by  combat  forces,  and  weighting  is  a  commonly  used 
technique  to  accomplish  this  consideration. 

To  overcome  this  problem,  and  to  better  capture  the  values  of  the  decision  makers,  BPS 
was  replaced  by  blue  personnel  casualties  (BPC),  denoted  as  'Pp,  as  a  measure  of  the  human  cost 
in  accomplishing  the  mission  of  the  force.  This  is  a  straight  forward  measure  that  is  not  a  ratio 
and  does  not  require  additional  weighting.  In  addition,  it  reflects  a  national  defense  goal  of 
winning  conflicts  quickly  with  few  personnel  lost. 

Clearly,  PER  and  BPS  or  BPC  are  not  independent.  Arguably  no  two  combat  MOE  are 
totally  independent,  and  no  single  MOE  is  sufficient  to  provide  enough  information  for  a 
complete  analysis.  These  MOE  were  understood  and  accepted  by  decision  makers  and  appeared 
to  highlight  important  performance  differences  between  systems  that  were  under  consideration. 

4.3  Response  Surface  Model 

The  MOE  described  above  are  proxy  measures  of  the  performance  of  an  entire  combat 
force  in  battle.  To  estimate  the  benefit  of  candidate  procurement  programs,  we  must  identify 
that  system's  contribution  to  the  effectiveness  of  the  force  with  respect  to  the  proxy  measures. 
The  most  common,  and  simplest,  method  of  determining  this  contribution  is  to  establish  a  base 
case  and  then  to  add  each  new  weapon  system  one  at  a  time,  measuring  the  changes  that  occur 
with  respect  to  the  particular  MOE.  These  changes  in  the  MOE  from  the  base  case  are  then 
taken  to  be  the  quantity  a  weapon  system  contributes  to  the  outcome  of  the  battle.  This  method 
ignores  the  fact  that  different  combinations  of  combat  systems  result  in  different  combat 
outcomes  and  that,  presumably,  several  of  the  candidate  procurement  programs  would  be  funded. 
Thus,  some  measurement  must  be  taken  that  considers,  at  least  in  some  aggregate  way,  the 
presence  of  other  new  systems  on  the  battlefield. 

The  ideal  solution  would  be  to  simulate  all  possible  combinations  of  candidate  systems. 
While  this  method  is  practical  for  situations  with  a  small  number  of  candidate  systems,  the 
number  of  combinations  grows  quickly.  For  example,  if  one  had  to  explore  every  combination  of 
40  different  systems,  the  number  of  potential  runs  would  be  2^^,  or  109.9  billion  simulation  runs. 

A  compromise  between  these  two  extreme  methods  is  to  apply  a  statistical  methodology. 
VAA's  model  is  based  on  a  general  linear  model  (GEM)  formulation  using  a  Plackett-Burman 
design.  See  Plackett  and  Burman  [19].  It  was  developed  to  perform  two  functions.  First,  the 
methodology  defines  a  design  matrix  which  specifies  the  combinations  of  candidate  systems  that 
are  to  he  simulated  in  each  run  of  the  combat  model.  This  design  plan  efficiently  maps  the  sample 
space  so  that  the  second  function,  the  building  of  a  regression  formula  that  can  be  used  to  predict 
the  response  of  the  CORBAN  model  with  respect  to  the  MOE,  can  be  performed. 

Plackett-Burman's  design  methodology  specifies  the  construction  of  the  design  matrix,  U. 
This  matrix  represents  a  map  of  all  the  independent  variables'  values  for  each  computer  run.  Each 
row  corresponds  to  a  specific  computer  run  and  each  column  corresponds  to  a  different  factor.  In 
the  case  of  VAA,  the  systems  being  considered  for  procurement  are  the  factors.  The  values  of  the 
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matrix  elements,  Ui,  are  either  1  or  0  which  represent  the  presence  or  absence  of  the  system  in  the 
simulated  battle,  respectively.  See  Mann  and  Loerch  [15]  for  a  complete  description  of  this  model. 
The  general  linear  model  is  of  the  form: 


"^3  =  Po  +  PlUl  +P2U2+P3U3  +  •••  +PmUm  » j  =  f.b,p, 

or  in  matrix  form, 

«Pj  -  up, 

where  'Tj  is  an  output  vector  for  MOE  j  of  the  model  run  results,  po  is  the  effect  of  the  base  case 
weapons.  Pi  through  Pm  are  the  effects  of  systems  1  through  m  in  the  excursions,  and  U  is  the 
design  matrix  of  binary  independent  variables  whose  construction  is  described  below.  The  values 
of  Pi  through  Pm  are  measures  of  the  average  contribution  of  candidate  systems  1  through  m  to  the 
effectiveness  of  the  force,  and  can  be  estimated  using  the  standard  formula  for  the  GLM.  See 
Neter,  et.  al.  [17].  This  model  isolates  only  the  presence  of  main  effects  without  considering  any 
interaction  effects. 

To  account  for  interactions,  careful  post-analysis  of  the  simulation  output  is  required  so 
that  combinations  of  systems  that  perform  particularly  well,  or  poorly,  are  identified.  Additional 
simulation  runs  are  added  to  the  original  design  so  that  extra  degrees  of  freedom  are  available  to 
analyze  specific  interactions.  Systems  that  are  determined  to  perform  in  a  synergistic  fashion  are 
reported  to  the  sponsors  and  are  subsequently  considered  together.  Conversely,  systems  that 
interfere  with  each  other  can  be  forced  to  be  procured  separately.  These  relationships  between 
systems  can  be  easily  handled  in  the  optimization  module. 

Note  that  in  each  excursion  a  different  combination  of  candidate  weapon  systems  is 
portrayed.  Care  was  required  to  ensure  that  the  proper  doctrine,  tactics,  and  methods  of 
employment  of  these  weapon  systems,  both  singly  and  in  combination,  were  represented.  This 
requirement  greatly  increased  the  workload  as  well  as  the  time  needed  to  complete  the  combat 
modeling  for  the  study. 


5.  EVALUATION  OF  OTHER  FACTORS 

A  shortcoming  of  previous  attempts  to  develop  a  program  tradeoff  methodology  was  that 
these  methodologies  concentrated  on  combat  measures  of  effectiveness  almost  exclusively.  These 
preceding  methodologies  (see,  for  example,  Shedlowski  [21]  )  did  not  accommodate  any  of  the 
additional  factors  which  decision  makers  felt  were  important  in  making  tradeoffs  between 
competing  weapon  systems.  In  this  section  we  discuss  the  inclusion  of  these  factors  in  VAA. 

5.1  Early  Work 

The  identification  and  definition  of  the  other  factors  that  are  important  in  evaluating  how 
much  a  system  contributes  to  the  effectiveness  of  the  force  are  performed  as  part  of  Issue 
Definition.  The  decision  as  to  which  factors  should  be  included  has  been  a  source  of  controversy 
in  the  past.  In  the  early  applications  of  VAA,  attempts  were  made  to  include  a  broad  range  of 
factors  whose  importance  was  readily  acknowledged.  Some  of  the  factors  considered  included: 
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(1)  Political  Risk.  This  criterion  is  used  to  evaluate  the  level  of  support  for  the 
program  across  the  political  spectrum. 

(2)  Operational  and  Technical  Risk.  This  measure  represents  a  subjective  evaluation 
of  the  probability  associated  with  a  program  meeting  all  of  its  stated  performance  criteria. 

(3)  Impact  on  Sustainability  (Combat).  This  criterion  is  aimed  at  the  burden  each 
system  places  on  the  logistical  system. 

(4)  Programmatics.  This  factor  measures  the  complexity  and  difficulty  of  contracting 
for  and  manufacmring  the  systems 

(5)  Asset  Versatility  and  Deployability.  This  criterion  measures  the  usefulness  of  the 
system  in  different  theaters  of  operations  and  different  combat  environments,  as  well  as  the  ease 
of  deploying  a  force  equipped  with  the  candidate  systems. 

(6)  Criticality  of  Need  As  Related  to  Current  Capability.  This  factor  attempts  to 
identify  systems  that  meet  some  critical  need,  or  overcome  some  important  shortcoming  in  force 
capability. 

These  criteria  were  viewed  to  be  entirely  subjective  and  measurement  was  performed  by 
using  a  survey  of  subject  matter  experts  from  the  Army  Staff.  The  original  plan  called  for  the  use 
of  the  Analytical  Hierarchy  Process  (AHP).  AHP  generates  measures  based  on  pairwise 
comparisons  between  the  candidate  systems  with  respect  to  each  criterion.  See  Saaty  [20]. 
However,  since  almost  6000  comparisons  would  have  been  needed  for  this  analysis,  this  survey 
technique  was  abandoned.  In  its  place,  an  individual  rating  was  made  for  each  system  on  each 
criterion  using  the  1-9  scale  from  AHP,  higher  being  better.  The  survey  reflected  a  consensus 
among  program  managers,  responsible  staff  officers  at  Army  Headquarters,  and  other 
organizations  as  appropriate. 

The  ratings  obtained  in  this  survey  were  then  used  as  part  of  the  overall  effectiveness 
scores  for  the  candidate  systems  for  use  in  the  objective  function  of  an  optimization.  The 
method  used  to  generate  the  overall  effectiveness  scores  is  discussed  in  the  next  section. 

This  approach  encountered  difficulties.  First,  two  of  the  identified  criteria  were  risk 
related,  representing  an  expert's  assessment  of  the  probability  that  the  system  would  be  fielded 
and  that  it  would  be  available  to  the  combat  commander  when  it  was  required.  In  theory,  a 
system  that  was  technically  infeasible  and  politically  impossible  to  support  but  performed  very 
well  in  the  combat  simulation  could  be  preferred  over  a  very  realistic  alternative  that  scored 
slightly  less  well  in  the  combat  simulation.  Second,  if  one  assumes  that  the  combat  simulations 
are  reliably  reflecting  the  performance  of  systems  on  the  battlefield,  and  the  combat  MOE  are 
reflecting  the  values  of  our  combat  leaders,  then  one  could  reasonably  assume  that  the  simulation 
is  capturing  "criticality"  of  the  need.  Third,  the  majority  of  the  "programmatics"  issues  were 
being  captured  in  the  cost  module  and  the  constraint  matrix  of  the  optimization. 

5.2  Subsequent  Efforts 

Based  on  this  early  experience  the  decision  was  made  to  eliminate  several  of  the 
evaluation  criteria,  as  well  as  the  subject  matter  expert  survey.  Although  the  modeling  effort  was 
thought  provoking  and  produced  some  good  insights  and  discussion,  not  much  contribution  was 
made  to  the  development  of  a  quantitative  measure  of  the  systems’  benefit  to  the  force. 


Consequently,  the  list  of  other  factors  considered  explicitly  was  paired  down  to  two  measurable 
attributes,  contribution  to  sustainability  and  contribution  to  deployability. 

5.3  Sustainability  Analysis 

The  Dictionary  of  US  Army  Terms  defines  sustainability  as  the  capability  of  the  force  to 
maintain  the  required  level  of  intensity  and  duration  of  combat  activity  to  achieve  the  force 
objectives  over  time.  Loosely  translated  this  means  that  the  force  is  well  sustained  when  the 
commander  has  sufficient  resources  to  do  what  he  wants  to  do.  The  decision  logic  of  the 
CORE  AN  model  includes  the  ability  to  track  the  usage  of  key  logistical  resources  for  each  unit 
represented  on  its  simulated  battlefield.  Thresholds  are  established  for  levels  of  supplies,  below 
which  the  unit  cannot  attack,  must  withdraw  when  confronted,  etc.  This  facility  of  the  model 
was  used  to  estimate  the  contribution  of  the  candidate  systems  to  the  sustainability  of  the  force. 

Discussions  were  held  with  expert  logistical  staff  officers  to  determine  which 
commodities  should  be  included  in  the  analysis.  Fuel  and  ammunition  supplies  were 
acknowledged  to  be  the  most  restrictive  battlefield  resources.  Therefore,  they  were  selected  as  the 
MOE  for  sustainability. 

The  measure  of  sustainability  was  computed  as  follows.  Let  tpq  be  the  total  time  combat 
unit  p  was  below  the  threshold  level  of  commodity  q  during  the  simulated  battle,  and  Tp  is  the 
total  time  unit  p  was  viable  during  the  battle.  Then,  if  P  is  the  set  all  combat  units,  the 
sustainability  of  the  force  with  respect  to  commodity  q  is  written  as 


X^pq 


peP 


This  quantity  is  collected  for  each  simulation  run  and  the  contribution  of  each  system  is 
determined  using  the  same  response  surface  methodology  that  was  used  to  evaluate  contribution 
to  combat  effectiveness. 

A  limitation  of  this  method  is  that  the  simulated  scenario  never  exceeds  seven  days  and  is 
typically  only  three  or  four  days  in  duration.  As  a  result,  the  force  is  not  badly  stressed  in  terms 
of  sustainment  in  that  time  and  making  strong  statements  about  contribution  of  individual 
systems  is  difficult.  Nevertheless,  the  observation  that  the  force  is  well  sustained  in  the  early 
parts  of  the  battle  is  an  important  one.  We  also  observed  that  systems  that  produced  enemy 
attrition  in  the  deep  battle,  such  as  some  missile  systems  with  long  ranges,  contributed  strongly 
to  sustainment  because  their  effective  employment  reduced  the  intensity  of  the  close  battle.  As  a 
result,  less  fuel  and  ammunition  were  expended. 

5.4  Deployability  Analysis 

Where  other  measures  of  effectiveness  and  contribution  could  be  estimated  using  the 
combat  simulation,  the  effect  of  fielding  the  candidate  systems  on  the  ability  to  deploy  the  force 


from  a  location  in  the  continental  United  States  to  an  overseas  location  could  not.  As  a  result, 
this  part  of  the  analysis  was  completed  by  experts  from  the  US  Army's  Military  Traffic 
Management  Command  (MTMC). 

Three  characteristics  of  the  candidate  systems  were  used  to  estimate  their  relative  impact 
on  the  deployability  of  the  force:  weight  in  short  tons,  surface  area  in  square  feet,  and  volume  in 
cubic  feet.  These  measures  included  not  only  the  end  item,  but  all  associated  items  of  equipment. 
This  is  often  referred  to  as  a  "unit  set".  Additionally,  information  on  scenario  and  force 
structure,  including  equipment  densities,  were  input  to  various  deployment  and  mobility  models 
that  are  used  by  MTMC  to  estimate  resource  requirements  for  force  deployments.  Estimates  of 
impact  are  made  by  comparing  the  required  resources  with  those  of  the  system  that  the  candidate 
system  replaced. 

This  procedure  was  performed  for  each  of  the  three  phases  of  the  movement,  the  so- 
called  “fort-to-port”  phase,  the  strategic  or  inter-theater  phase,  and  the  intra-theater  or  “port-to- 
foxhole”  phase.  Resources  that  were  considered  included  quantities  and  types  of  railcars  and 
other  surface  transportation  assets  like  flatbed  trailers  and  heavy  equipment  transporters.  Of 
particular  interest  was  whether  or  not  the  equipment  that  was  to  be  transported  by  air  required 
the  use  of  the  larger,  but  less  numerous,  C5A  cargo  aircraft.  Since  C5A’s  are  a  very  scarce  and 
important  strategic  mobility  resource,  the  requirement  of  their  use  for  a  new  item,  when  its 
predecessor  system  could  be  transported  using  the  more  abundant  C141  aircraft,  would  represent 
a  significant  negative  impact  on  the  deployability  of  the  force. 

A  measure  of  contribution  to  the  deployability  of  the  force  for  each  leg  or  phase  was 
developed  by  making  the  evaluation  described  above,  weighting  the  assets  required  based  on 
existing  and  forecast  availability  and  capability,  and  then  summing  the  scaled  and  weighted  scores 
for  each  system.  An  overall  measure  was  then  computed  by  summing  the  weighted  scores  for  all 
legs  of  the  deployment.  These  weights  are  also  based  on  the  capabilities  that  exist  to  execute  the 
movement  phases. 


6.  EFFECTIVENESS  INTEGRATION 

The  purpose  of  the  effectiveness  integration  module  is  to  derive  a  set  of  system 
"scores"  that  reflects  the  relative  contribution  of  the  potential  weapon  systems,  or  acquisition 
programs,  to  the  performance  of  the  Army  in  combat.  The  scores  are  used  to  form  the 
coefficients  in  the  objective  function  of  the  optimization  module.  The  reliability  of  any 
acquisition  strategy  that  is  recommended  by  the  mathematical  function  relies  on  the  quality  of 
these  coefficients.  These  coefficients,  then,  must  be  developed  in  a  rational  and  consistent 
way  to  be  meaningful.  In  particular,  the  individual  measures  or  attributes  that  make  up  the 
overall  scores  and  the  weights  on  the  importance  of  these  attributes,  must  reflect  the  values  of 
the  Army  leadership  and  must  be  consistent  with  Army  objectives. 

6.1  Early  Work 

Initial  efforts  to  integrate  the  heterogeneous  measures  of  effectiveness  into  a 
homogeneous  measure  applied  the  Analytic  Hierarchy  Process  (AHP)  [20]  to  produce  the 


criteria  weights  and  a  multi-attribute  decision  algorithm  called  the  Technique  for  Ordered 
Preference  by  Similarity  to  an  Ideal  Solution  (TOPSIS)  [9, 25]  to  compute  the  overall  system 
scores.  Research  into  the  behavior  of  the  algorithms  as  they  were  applied  to  this  problem 
indicated  that  spurious  results  could  occur.  See  Maxwell  and  Buede  [16].  Moreover, 
research  into  the  algorithms  in  general  indicated  that  the  results  being  experienced  in  the  VAA 
application  were  symptomatic  of  more  general  shortcomings.  These  are  discussed  by  Buede 
and  Maxwell  [2],  Dyer  [5],  and  Kloeber  [11].  Because  of  these  difficulties  the  approach 
evolved  into  the  model  described  below. 

6.3  The  Current  Model 

There  are  two  key  elements  to  any  model  that  must  be  separated  and  addressed 
explicitly,  if  a  model  is  to  provide  meaningful  insight.  These  are  issues  of  fact  and  issues  of 
value.  For  further  discussion  of  this  separation  see  Keeney  [10].  Sections  Four  and  Five 
previously  identified  and  discussed  five  criteria  that  are  being  used  as  proxies  for  contribution 
of  a  weapon  system  to  the  overall  performance  of  the  Army.  Section  Two  identified  two 
time  frames  and  two  scenarios  that  are  to  be  considered  in  developing  the  relative  contribution 
of  systems.  These  data  provide  the  "factual"  information  that  will  be  used  to  derive  the 
coefficients.  For  example,  adding  system  X  to  the  force  mix  increases  the  expected  FER  by  .5, 
from  1.0  to  1.5  in  the  far  term  contingency  scenario.  Similarly,  the  substitution  of  system  Y 
for  an  old  system  reduces  the  lift  requirement  for  the  modeled  force  by  approximately  70  C- 
141  sorties,  or  approximately  one  percent. 

The  factual  outcomes  described  above  will  either  be  true,  or  not,  or  we  may  never 
know  because  the  situation  does  not  occur.  How  senior  leaders  feel  about  improvements 
within,  and  trade-offs  among  these  measures  are  issues  of  value.  There  is  no  "right"  or 
"wrong"  answer,  and  these  are  the  judgments  for  which  these  leaders  are  responsible. 

Analysis  must  seek  to  identify  and  apply,  as  accurately  as  possible,  the  values  of  the  Army's 
senior  leaders.  Where  possible  consensus  should  be  sought.  Where  consensus  is  not 
possible,  robust  sensitivity  analysis  should  seek  out  the  implications  of  the  value  differences 
with  respect  to  the  alternatives  under  consideration.  VAA  addresses  this  goal  directly. 

In  the  most  current  iteration  of  the  analysis,  the  MOE  were  arranged  hierarchically  as 
indicated  in  Figure  2.  The  figure  points  out  the  fact  that  the  overall  "score"  for  a  system  in  a 
given  time  frame  is  a  function  of  the  previously  identified  measures.  This  model  then 
provides  a  context  within  which  to  assess  the  preferences  of  senior  leaders.  In  this  instance 
of  the  analysis  approximately  ten  general  officers  and  senior  civilians  were  interviewed. 

The  actual  interview  with  the  senior  leaders  was  a  structured  interview  using  a 
derivative  of  an  elicitation  technique  called  Simple  Multi-attribute  Rating  Technique 
(SMART)  developed  by  Edwards  [6].  This  method  employs  an  approximation  technique 
that  has  been  successfully  applied  in  practice  by  VAA  team  members.  It  was  chosen  for 
three  reasons.  First,  it  has  proven  to  give  reasonable  weight  approximations  for  multi¬ 
attribute  problems  with  considerably  smaller  labor  investments  than  more  rigorous 
techniques.  Second,  since  senior  leader  time  is  a  valuable  commodity,  analysts  needed  to  be 
in,  understood  by  the  decision  maker,  elicit  their  input,  and  out  of  the  office  in  under  an  hour. 


Finally,  the  elicitation  session  needed  to  be  highly  interactive  and  allow  for  immediate 
feedback  to  the  decision  maker.  This  interaction  assured  that  the  decision  maker  and  analyst 
were  communicating  clearly  and  served  to  build  senior  leader  confidence  in  the  analysis 
process  by  "getting  their  hands  dirty".  Commercially  available  software  made  this  interaction 
possible. 


Figure  2.  Value  Added  Analysis  Hierarchy 

In  order  to  complete  an  interview  using  SMART  an  analyst  must  clearly  identify  the 
attributes,  as  well  as  the  range  those  attributes  vary  over  in  the  given  decision  context.  This 
specificity  appropriately  frames  the  judgments  for  the  decision  maker  and  allows  for  better 
informed  trade-off  or  ranking  decisions  than  other  methods.  This  change  resulted  in  a  much 
greater  degree  of  consensus  among  decision  makers  regarding  the  relative  importance  of 
attributes  than  previously  employed  multi -attribute  methods. 

6.4  Integration  Module  Outputs 

As  previously  identified,  the  most  important  output  of  the  Integration  Module  is  a 
set  of  system  coefficients.  The  coefficients  are  not,  however,  the  same  value  that  is 
produced  by  the  multi-attribute  function.  It  is  derived  from  that  value  based  on  two  relevant 
factors.  First,  the  contribution  to  the  force  that  is  estimated  by  the  simulation  and  experts 
considers  the  system  as  part  of  a  contingency  corps.  The  potential  systems  are  assigned  to 
appropriate  units  in  varying  quantities  based  on  appropriate  doctrine  and  mission  objectives. 
These  quantities  must  be  adjusted  so  they  are  consistent  with  incrementing  units  being 
considered  by  the  mathematical  program.  Additionally,  the  objective  function  requires  a  ratio 
measure.  This  is  derived  by  estimating  the  performance  of  the  baseline  force.  This  baseline  is 
then  subtracted  from  the  estimated  performance  of  the  force  with  the  system.  The  difference 
is  the  marginal,  or  incremental,  improvement  of  the  force’s  performance. 


7.  COST  ANALYSIS 


The  purpose  of  the  cost  analysis  module  is  twofold.  First,  accurate  system  procurement 
costs  must  be  estimated,  including  research,  development,  test,  and  evaluation  (RDT&E)  costs; 
fixed  production  costs;  and  variable  production  costs.  These  costs  are  necessary  for  conducting 
the  optimization  which  finds  the  mix  of  systems  that  maximizes  the  effectiveness  of  the  force 
subject  to  constraints  on  the  Research,  Development,  and  Acquisition  (RDA)  budget.  Second, 
given  optimized  quantities  of  procured  systems,  estimates  must  be  computed  of  the  other 
aspects  of  the  life  cycle  costs  of  the  system.  These  aspects  include  fielding  costs,  sustainment 
costs,  and  facilities  costs.  Once  the  various  components  of  the  life  cycle  costs  are  computed, 
they  must  be  made  available  in  an  easily  accessible  form  so  that  information  regarding  the  costs  of 
the  candidate  systems  can  be  analyzed. 

An  important  aspect  of  estimating  the  procurement  costs  associated  with  the  candidate 
systems  is  determining  whether  or  not  a  significant  relationship  exists  between  the  unit  cost  of  a 
system  and  the  quantity  procured.  For  many  of  the  candidate  systems,  particularly  the 
developmental  systems  that  involve  new  technology,  this  cost-quantity  relation  is  significant  and 
nonlinear.  As  such,  this  relationship  must  be  considered  and  accounted  for  to  ensure  accurate 
results.  For  the  purposes  of  this  study,  the  cost-quantity  relationship  reflects  economies  of  scale 
in  terms  of  materials  and  labor,  as  well  as  "learning"  on  the  part  of  the  production  labor  force. 

Although  there  are  many  mathematical  models  of  these  so-called  learning  curve  costs  (see 
Dutton,  et.  al.  [4]  for  a  review  of  about  300  articles  that  discuss  learning  curves),  the  so-called 
power  or  “exponential”  model  was  chosen  to  describe  the  cost-quantity  relationships  in  the 
VAA  Study.  This  model  is  commonly  used  in  defense  and  industrial  cost  estimation  and  was 
adopted  in  VAA  to  maintain  consistency  between  cost  estimates  used  in  this  study  effort  and 

others  provided  for  the  purpose  of  staff  planning  and  budgeting.  The  model  has  the  following 
form: 

c  =  Ay"’’, 

where 

c  =  cost  of  the  y*  unit  of  a  system  produced, 

A  =  the  notional  cost  of  the  first  unit  produced, 
b  =  cost-quantity  slope  coefficient, 
y  =  accumulated  number  of  items  produced. 

Notice  that  this  function  is  nonlinear  as  is  the  case  in  almost  all  representations  of  learning 
effects. 

Average  unit  procurement  costs  are  used  for  the  systems  that  exhibit  no  appreciable  learning 
behavior.  Fixed  costs,  i.e.,  costs  that  do  not  depend  on  the  quantity  of  an  item  procured,  are  also 
mcluded.  These  costs  include  RDT&E  expenditures  and  fixed  production  costs.  Cost  data  was 
obtained  from  multiple  sources  including  the  various  program  managers  that  are  responsible  for  the 
candidate  procurement  programs. 

Following  the  computation  of  a  recommended  system  mix  for  procurement,  together  with 
optimal  procurement  quantities,  the  life  cycle  costs  for  the  programs  are  estimated  using  the  Life 
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Cycle  Cost  Model  (LCCM).  This  model,  implemented  on  a  spreadsheet,  uses  standardized  cost 
definitions  (see  DCA-P-92  (R)  [22])  and  previously  determined  cost  estimating  relationships  to 
compute  the  cost  of  fielding,  sustaining  ,  and  providing  adequate  facilities  for  the  system.  The 
LCCM  automates  the  recosting  of  the  life  cycle  costs  based  on  a  new  set  of  quantities,  thus 
providing  a  tool  for  conducting  cost  related  "What  If  s". 

8.  OPTIMIZATION 

The  actual  cost  effectiveness  analysis  contained  within  the  Value  Added  Methodology  is 
performed  in  the  optimization  module.  In  the  current  version  of  VAA,  the  final  determination  of  a 
recommended  acquisition  strategy  is  based  on  the  acquisition  cost  of  a  system,  the  combined 
effectiveness  value  which  includes  both  the  combat  and  other  factor  values,  the  force  stmcture 
requirement  for  the  systems  to  fill  the  force,  the  capability  of  industry  to  produce  the  systems,  and 
budgetary  ceilings.  Furthermore,  since  the  current  budgetary  environment  has  resulted  in  more 
programs  being  available  for  procurement  than  there  are  funds  available  to  pay  for  them,  it  was  clear 
that  some  method  of  identifying  systems  that  should  be  left  unfunded  was  needed.  Also,  some 
programs  because  of  their  operational  requirements,  needed  to  be  linked  together  in  such  a  way  that 
the  model  would  only  recommend  that  operationally  logical  sets  be  procured.  The  VAA 
optimization  model  was  developed  as  a  mixed  integer  program  capable  of  handling  the  factors 
discussed  above. 

The  objective  of  the  model  is  to  suggest  a  mix  of  systems  for  procurement  that  will  be  as 
effective  as  possible  in  combat,  subject  to  a  total  budget  authority  ceiling.  The  values  developed  in 
the  effectiveness  integration  module  are  used  to  create  a  single  measure  of  a  system's  worth  for  each 
year  in  the  planning  horizon,  reflecting  all  the  measures  of  effectiveness  for  each  program,  weighted 
by  the  senior  leader  survey  process,  and  scaled  to  a  per  item  value.  The  objective  of  the 
mathematical  program  is  to  maximize  the  sum  of  these  system  values.  This  is  done  by  constraining 
the  model  to  an  overall  Total  Obligation  Authority  (TO A)  which  represents  a  yearly  budget  ceiling,  a 
particular  Army  force  structure  which  determines  the  quantities  of  the  new  systems  required,  and  the 
ability  of  the  industrial  base  to  produce  the  systems  as  defined  by  start  and  stop  dates  and  aimual 
production  quantities.  Each  of  these  constraints  is  computed  by  year  for  both  the  POM  years  and 
the  EPP,  a  total  of  fifteen  years.  The  formulation  of  the  model  can  be  summarized  as  follows. 
Maximize:  Force  effectiveness, 

Subject  to:  Budget  ceiling. 

Production  limitations. 

Force  structure  requirements,  and 
Decision  constraints, 

where  the  decision  variables  for  this  model  are  the  quantities  of  each  system  procured  in  each  year. 

As  mentioned,  available  funds  are  never  sufficient  to  procure  all  candidate  systems,  even  at 
the  lowest  allowable  quantities.  Also,  quantities  must  be  constrained  such  that,  if  a  program  is 
recommended,  the  requisite  quantities  will  be  procured  to  meet  force  structure  requirements  and  to 
not  exceed  the  capabilities  of  the  existing  or  planned  production  facilities.  Thus  a  mixed  integer 
programming  formulation  is  introduced  where  the  variable  ui  is  defined  as  follows: 


(1,  if  candidate  system  i  is  recommended  for  procurement, 
0,  otherwise. 


So  the  constraints  on  the  quantities  produced  take  the  following  form: 


where 


P  ■  u-  <  X  -  <  P  u- 

^  minjj  “i  -  ^ij  -  ^maxjj  *^1  ’ 

xjj  =  the  quantity  of  system  i  procured  in  year  j. 


and  Pminij  aiid  Pmaxjj  constrain  the  yearly  production  quantity.  Note  here  that  if  the  system  is  not 
recommended  for  procurement,  the  value  of  Ui  is  zero  and  the  quantities  will  also  be  constrained  to  be 
zero.  Similar  constraints  are  iinplemented  on  the  total  procurement  of  each  system.  The  binary 
variable  Ui  is  also  used  to  assess  fixed  costs  in  the  budget  constraint. 

Use  of  the  binary  variables,  Ui ,  is  made  to  constrain  the  model  to  implement  previously  made 
decisions.  These  decisions  are  implemented  in  the  model  by  constraining  linear  functions  of  the 
vector  of  U;  variables,  fk(u).  For  example,  if  the  decision  is  made  to  procure  a  certain  system  i,  then 
the  constraint  ui  =  1  is  included  in  the  formulation.  If  two  systems  are  related  in  their  procurement 
such  that  system  2  is  procured  only  if  system  1  is  procured,  a  constraint  of  form  ui  -  U2  =  0  would 
be  included.  These  decision  constraints  added  flexibility  to  the  model  and  allowed  the  evaluation  of 
previously  made  or  contemplated  decisions  regarding  force  effectiveness  and  opportunity  costs  with 
respect  to  the  procurement  of  other  systems. 

The  formulation  of  the  mathematical  programming  model  is  expressed  as  follows: 


Max  X  X 

i=ljeJi 

m 

s.t.  ^(qjXij  +  CijUi)<Bj;  j=l,...,n; 
i=l 

PminijUi  ^Xjj  <PnjaxijUi;  i  =  1,...,  m;  j  e  Jj; 
jeJ; 

(l-Pi)xij-i  <Xij  <  (1+Pi)xij_i;  i  =  1,...,  m;  je  Ji;  C-l)e  Ji; 

Xij>0;  i  =  l,...,m;  jeJ,; 
uj  €{0,l},  i  =  1,...,  m. 


where 


n  =  number  of  years  in  the  planning  horizon, 

Ji  =  {j  •  system  i  can  be  procured  in  year  j,  j  e  {1,...,  n}}, 
m  =  number  of  candidate  procurement  programs, 
xjj  =  quantity  of  system  i  produced  in  year  j, 

Vy  =  system  effectiveness  coefficient  of  system  i  for  year  j, 
qj  =  unit  variable  cost  of  system  i  in  year  j, 

Cjj  =  fixed  cost  of  system  i  in  year  j, 

Bj  =  RDA  funds  available  in  year  j, 

Pmiiijj  =  minimum  allowable  production  quantity  of  system  i  in  year  j, 

Pmaxjj  =  maximum  allowable  production  quantity  of  system  i  in  year  j, 

Rminj  =  minimum  quantity  of  system  i  required  by  the  target  force  Structure, 

Rmaxj  =  maximum  quantity  of  system  i  required  by  the  target  force  Structure, 

Pi  =  allowable  percentage  deviation  from  previous  year  production  quantity, 
djj  =  minimum  value  for  decision  constraint  k, 

K  =  number  of  decision  constraints, 

=  maximum  value  for  decision  constraint  k, 

=  decision  constraint  function  of  the  vector  of  Uj  variables. 

Another  use  for  binary  variables,  not  shown  above,  arose  from  the  requirement  to  include  the 
nonlinear  learning  curve  costs  in  the  optimization,  as  described  in  the  section  above  on  cost  analysis. 
A  piecewise  linear  approximation  was  used  to  represent  these  costs,  precipitating  the  need  for 
another  class  of  binary  variables.  The  development  of  this  approximation  is  discussed  in  detail  by 
Loerch  [14]. 

This  model  was  implemented  using  the  Optimization  Subroutine  Library  (OSL),  on 
an  IBM  RISC  6000  work  station.  A  matrix  generator  was  written  in  FORTRAN.  The  mixed 
integer  program  had  approximately  4,000  rows  with  3,000  variables,  of  which  about  800  were 
binary  integers,  and  5,500  non  zero  elements.  The  run  time  for  this  model  was  between  10 
and  60  minutes  of  CPU  time  on  the  IBM  RISC  6000  590,  with  an  average  of  20  minutes. 

The  model  was  successfully  used  to  assist  the  Army  Staff  in  evaluating  the  various 
alternative  weapon  system  mixes  considered  for  procurement.  The  model  was  particularly  useful  in 
identifying  the  years  in  which  budget  constraints  were  extremely  tight  with  respect  to  planned 
production  campaigns,  suggesting  modifications  that  could  be  made  to  the  proposed  programs.  The 
model  was  also  useful  in  identifying  systems  that  were  excluded  from  the  solution  when  other 
systems,  or  combination  of  systems,  were  forced  to  be  procured.  This  capability  gave  the  Army 
leadership  a  window  into  the  cost  of  their  decisions  as  they  related  to  system  tradeoffs. 

The  use  of  the  optimization  model  for  the  purpose  of  comparing  alternative  strategies  and 
evaluating  tradeoffs  far  outweighed  the  fact  that  a  particular  recommended  system  mix  was  produced. 
The  feedback  we  received  from  decision  makers  indicated  that  these  insights  were  much  more  useful 


in  the  decision  making  process  than  specific  recommendation  since  they  tended  to  be  more  globally 
applicable  and  because  the  situation  with  respect  to  decisions  that  were  made  and  funding  levels  was 
constantly  changing.  For  this  reason,  the  use  of  this  optimization  model  was  iterative,  as  more  and 
more  “What-if  ’  questions  were  asked. 

9.  THEATER  FORCE  EVALUATION 

The  need  arose  to  have  the  capability  to  evaluate  the  acquisition  strategy  recommendation 
made  using  the  optimization  module  in  the  context  of  theater  operations  and  over  various  force 
structures. 

The  previously  described  modules  in  the  methodology  dealt  solely  with  effectiveness  of 
modernization  systems  in  the  context  of  the  operations  of  a  fully  modernized  Army  corps.  The 
force  structure  of  the  entire  Army  was  given  and  assumed  fixed.  For  example,  a  12  division 
"Base  Force"  was  assumed  during  the  conduct  of  the  study  and  the  quantity  required  to  be 
procured  for  each  modernization  system  was  specified  by  the  sponsor  as  a  function  of  that  force. 

As  the  budget  building  process  progressed,  however,  consideration  was  given  to  the 
reduction  of  the  base  force  to  less  than  12  divisions.  The  question  then  needed  to  be  answered  as 
to  the  capabilities  of  the  reduced  force  when  some  of  the  monetary  savings  achieved  from  the 
force  reduction  are  applied  to  increased  modernization.  This  question  can  only  be  answered  by 
analyzing  the  force  at  the  theater  level. 

The  procedure  was  simple.  A  system  mix  and  acquisition  strategy  was  developed  using  the 
optimization  module,  with  the  budgetary  constraint  relaxed  to  take  into  account  the  additional 
funds  made  available  through  the  reduction  in  force,  and  with  the  required  quantities  of 
equipment  changed  to  reflect  the  reduced  force  structure.  The  equipment  recommended  for 
procurement  was  then  distributed  to  the  units  in  the  designated  force,  and  this  force  was 
simulated  in  an  agreed  upon  scenario.  The  results  were  then  compared  to  those  from  the  original 
base  force. 

The  methodology  was  limited  in  the  sense  that  the  theater  simulation  was  much  more 
aggregate,  so  some  of  the  systems  whose  contribution  could  be  measured  at  the  corps  level  had 
no  measurable  effect  at  the  theater  level.  Also,  the  procedure  was  fairly  manpower  intensive,  so 
only  a  few  alternative  force  structures  could  be  evaluated  within  the  required  time  frame. 

Nevertheless,  the  results  proved  interesting  and  insightful,  and  this  part  of  the  methodology 
was  made  a  permanent  part  of  the  VAA  methodological  framework. 

10.  RESULTS 

Figures  3  and  4  provide  an  example  of  analysis  output  provided  to  senior  Army  decision 
makers.  Figure  3  is  a  so  called  "X  chart"  which  shows  which  candidate  programs  were 
recommended  for  procurement  and  which  ones  were  not.  Systems  are  categorized  and  listed  in 
order  of  Battlefield  Operating  System,  namely  Air  Defense  (AD),  Aviation  (AVN),  Command 
and  Control  (C2),  Engineer  (ENGR),  Indirect  Fire  (IDF),  Intelligence  and  Electronic  Warfare 
(lEW),  Logistics  (LOG),  and  Maneuver  (MVR).  An  X  in  the  chart  indicates  that  the  particular 
system  was  recommended  in  the  optimization  module  for  a  particular  excursion  and  a  blank 
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indicates  that  it  was  not.  Typically,  several  excursions  were  performed  in  each  analysis.  Each 
excursion  examined  a  different  set  of  conditions.  These  conditions  included  the  level  of  funding 
for  RDA  programs  and  different  decisions  that  had  been  made  or  were  contemplated  regarding  the 
various  candidate  programs.  The  tradeoffs  associated  with  each  mix  may  be  seen  by  comparing 
across  the  columns  and  identifying  those  systems  which  come  into  and  leave  the  recommended 
list  as  the  conditions  change.  We  note  that  the  typical  set  of  solutions  will  be  characterized  by 
several  systems  that  are  always  in  the  recommended  list,  several  that  are  never  recommended,  and 
still  others  that  are  sometimes  in  and  sometimes  out  of  the  recommended  list.  Thus  the  decision 
makers  can  assess  the  relative  desirability  of  the  various  programs  under  a  given  set  of  conditions. 

Figure  4  provides  an  example  of  graphical  displays  of  the  combat  effectiveness  of  the 
various  mixes  of  systems  recommended.  These  estimates  are  made  using  the  response  surface 
models  developed  in  the  Combat  Effectiveness  module.  Ideally,  the  combat  simulation  would  be 
re-run  with  the  various  mixes  portrayed  explicitly.  However,  the  results  of  these  analyses  were 
usually  needed  within  a  24-48  hour  period,  making  it  impossible  to  re-run  the  combat 
simulations.  As  such,  the  response  surface  model  became  an  important  tool  for  performing  quick 
response  analyses.  This  output  is  extremely  important  for  the  evaluation  of  the  recommended 
system  mixes  because  it  shows  the  effectiveness  of  the  force  in  terms  that  the  decision  maker  can 
understand.  We  have  found  that  reporting  differences  in  the  objective  for  various  excursions  is 
not  meaningful.  The  individual  measures  that  are  used  to  build  the  objective  function  must  be 
broken  out  and  shown  separately.  Similar  displays  are  produced  for  all  the  individual  measures. 

Typically,  no  one  recommended  system  list  from  the  excursions  dominates  the  others  in 
all  measurement  criteria.  Such  is  the  case  in  figure  4  which  shows  the  fractional  exchange  ratio. 
Insight  can  be  drawn,  however,  from  the  examination  of  the  recommended  systems  and  their 
performance  in  various  scenarios.  Note  also  that  the  results  of  a  base  case  run  in  which  no 
modernization  is  portrayed  is  also  shown.  This  run  is  used  to  show  the  contribution  of  the 
modernization  program  as  a  whole. 

Not  shown  here,  but  provided  to  the  decision  makers  for  each  excursion,  is  a  table 
showing  the  yearly  procurement  quantities  as  well  as  a  breakdown  of  the  research,  development, 
test  and  evaluation  costs,  the  fixed  production  costs,  and  the  variable  production  costs  for  the 
systems  on  the  recommended  procurement  list. 
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Recommended  URR  Systems 


VAA  Systems  By 


AD  I  AD  Veh 


MSL  1 


Radar  1 


Radar  2 


MSL  2 


AVN  iHelo  1 


Helo  2 


HeJo  3 


Helo  4 


C2  ICC  1 


CC  2 


CC  3 


CC  4 


CC  5 


CC  6 


BOS 


Air  Defense  Vehicle 


Air  Defense  Missile  1 


Air  Defense  Radar  1 


Air  Defense  Radar  2 


Air  Defense  Missile  2 


Helicopter  Upgrade  1 


£xc  l|£xc  2|£xc  3  £xc  4  £xc  5 


X  X 


Mine  1 


Mine  2 


Mine  3 


Ammo  1 


Radar  3 


Radar  4 


MSL  3 


MSL  4 


Arty  Veh  1 


Ammo  2 


lEW  iSensor  1 


Sensor  2 


Radar  5 


I£W  1 


lEW  2 


Log  Veh  1 


Truck  1 


Truck  2 


2 


Armor  1 


MVR  Sys  1 


MSL  3 


Armor  2 


Armor  3 


Ammo  3 


Ammo  4 


New  Helicopter  3 


New  Helicopter  4 


Command  and  Control  System  1 


Command  and  Control  System  2 


Command  and  Control  System  3 


Command  and  Control  System  4 


Command  and  Control  System  5 


Command  and  Control  System  6 


Engineer  Equipment  1 


Engineer  Vehicle 


Mine  System  1 _ 

Mine  System  2 _ 

Mine  System  3 
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Figure  3.  X-Chart  -  Recommended  Systems  Lists  by  Excursion 
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I  e  Scenario  1  Near  Term 
I  ■  Scenario  1  Far  Term 

□  Scenario  2  Near  Term 

□  Scenario  2  Far  Term 


Figure  4.  Combat  Performance  of  Recommended  System  Mix 
11.  SUMMARY  AND  CONCLUSIONS 

The  primary  result  of  the  development  portion  of  the  study  was  the  definition  of  a  flexible  and 
rational  methodology  to  support  the  POM  process  and  Army  program  development.  The 
methodology  developed  during  this  effort  provided  a  way  of  capturing  the  subjective  elements  used 
by  decision  makers  in  conducting  tradeoffs  between  alternatives  through  the  use  of  judgment  weights 
This  phase  of  the  study  also  provided  some  important  insights  into  the  use  of  aggregated 
effectiveness  models,  new  eosting  approaches,  and  optimization  techniques  which  were  built  upon 
later. 

As  the  VAA  methodology  was  used  to  support  decision  making,  the  following  observations 
were  made. 

(1)  Each  module  of  the  methodology  provides  important  information  and  insight  to  the 
decision  makers.  For  example,  the  response  surface  methodology  provided  important  insights  into 
contribution  of  different  weapons  systems  to  the  performanee  of  the  force  in  combat. 

(2)  The  data  collection  effort  and  subsequent  building  of  the  VAA  databases 
resulted  in  a  comprehensive  set  of  data  that  was  not  available  elsewhere. 

(3)  The  purpose  of  VAA  is  to  build  the  capability  to  do  quick  turnaround  "What  If 
analyses.  To  meet  this  goal,  however,  requires  eight  months  or  more  of  preliminary  effort. 

Once  this  preliminary  work  is  completed,  new  solutions  based  on  the  questions  and 
constraints  imposed  by  the  decision  makers  can  be  computed  very  rapidly. 

The  use  of  Value  Added  Analysis  provides  decision  makers  at  the  Department  of  the 
Army  level  with  a  tool  to  quickly  evaluate  programming  and  budgeting  decisions  in  the  area  of 
equipment  modernization.  Additional  work  is  needed  to  expand  the  methodology  to  include 
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expenditures  in  other  areas  such  as  force  structure,  operations  and  training,  personnel,  and 
facilities. 


Acknowledgments 

All  three  authors  would  like  to  acknowledge  the  support  of  Mr.  E.B.  Vandiver  III,  the 
Director  of  the  US  Army  Concepts  Analysis  .  He  is  a  long-standing  leader  of  Army  Analysts, 
who  has  encouraged  creative  efforts,  such  as  this  .  We  thank  Professor  Rick  Rosenthal  of  the 
Naval  Post  Graduate  School  for  sharing  his  ideas  concerning  optimization,  decision  support  in 
general,  and  this  paper.  We  are  grateful  to  the  many  other  members  of  the  VAA  team  for  their 
hard  work  and  creativity.  They  are  to  numerous  to  identify  here,  and  collectively  they  are  the 
reason  for  any  success  VAA  may  have  achieved. 

REFERENCES 

[1]  Anderson  Michael  R.,  "Prioritizing  Modernization  Actions  for  the  Army 
Modernization  Memorandum  (AMM)  and  Long  Range  Army  materiel  Requirements 
Plan  (LRAMP)",  Proceedings  of  the  60th  Symposium  of  the  Military  Operations 
Research  Society,  23-25  June  1992. 

[2]  Buede,  D.M.  &  Maxwell,  D.T.  Rank  Disagreement:  A  Comparison  of  Multi-Criteria 
Methodologies,  Journal  of  Multi-Criteria  Decision  Analysis,  Vol.  4,  pp.1-21,  1995. 

[3]  Coyle,  R.  G.,  "The  Optimisation  of  Defence  Expenditure",  European  Journal  of 
Operational  Research,  56,  3,  February  10,  1992,  304-18. 

[4]  Dutton  John  M,  Annie  Thomas,  and  John  E.  Butler,  "The  History  of  Progress 
Functions  as  a  Management  Technology",  Business  History  Review,  58, 2  (Summer 
1984),  202-33. 

[5]  Dyer,  James  S.,  "Remarks  on  the  Analytical  Hierarchy  Process",  Management 
Science,  36,  3,  249-58,  March  1990. 

[6]  Edwards,  W.  &  Barron,  H.  SMARTS  and  SMARTER:  Improved  Simple  Methods  for 
Multi-attribute  Utility  Functions,  Organizational  Behavior  and  Decision  Processes,  In 
Press. 

[7]  Hall,  Nicholas  G.,  John  C.  Hershey,  Larry  G.  Kessler,  and  Craig  Stotts,  "A  Model  for 
Making  Project  Funding  Decisions  at  the  National  Cancer  Institute",  Operations 
Research,  40,  6,  November-December  1992, 1040-1052. 

[8]  Helmbold,  Robert  L.,  "Rates  of  Advance  in  Historical  Land  Combat  Operations,"  to 
appear  in  Naval  Research  Logistics  Special  Issue  on  air-land-naval  warfare  models. 


23 


[9]  Hwang,  Ching-Lai  and  Y oon,  Kwangusn,.  Multiple  Attribute  Decision  Making, 
Methods  and  Applications,  Berlin,  Springer-Verlag,  1981. 

[10]  Keeney,  Ralph,  Value  Focused  Thinking,  Harvard  University  Press,  Cambridge  MA, 
1993 

[11]  Kloeber,  Jack  M.  Jr.,  A  Critical  Analysis  of  the  Technique  for  Order  Preference  by 
Similarity  to  Ideal  Solution  (TOPSIS),  Department  of  Systems  Engineering,  United 
States  Military  Academy,  Report  No.  91-1,  July  1991. 

[12]  Koury,  Robert  R.,  Army  Program  Value  Added  Analysis  90-97  (VAA  90-97)  Study, 
CAA-SR-91-9,  US  Army  Concepts  Analysis  Agency,  Bethesda,  MD,  1991. 

[13]  Koury,  Robert  R.  and  Andrew  G.  Loerch,  Army  Program  Value  Added  Analysis  94- 
99  (VAA  II)  Study,  CAA-SR-92-10,  US  Army  Concepts  Analysis  Agency, 

Bethesda,  MD,  1992. 

[14]  Loerch,  Andrew  G.,  Value  Added  Linear  Optimization  Of  Resources  (VALOR), 
CAA-TP-92-1,  US  Army  Concepts  Analysis  Agency,  Bethesda  MD,  March  1992. 

[15]  Maim,  William  F.  Ill  and  Andrew  G.  Loerch.  "Response  Surface  Methodology  for 
Value  Added  Analysis",  Proceedings  of  the  Thirty  Seventh  Conference  on  the  Design 
of  Experiments  in  Army  Research,  Development  and  Testing,  February  1992. 

[16]  Maxwell,  D.T.  &  Buede,  D.M.  Multi-Attribute  Decision  Algorithms:  A  Comparison  of 
TOPSIS  and MAUT,  Submitted  to  Military  Operations  Research,  February  1994 

[17]  Neter,  J.,  Wasserman  W.  and  Kutner,  M,  Applied  Linear  Statistical  Models,  Irwin, 
Homewood,  IL,  1985. 

[1 8]  Peereboom,  James  P.,  William  A.  Buehring,  and  Timothy  W.  Joseph,  "Selecting  a 
Portfolio  of  Environmental  Programs  for  a  Synthetic  Fuels  Facility",  Operations 
Research,  37,  5,  September-October  1989,  689-699. 

[19]  Plackett,  R.L.  and  J.P.  Burman,  "The  Design  of  Optimal  Multifactorial  Experiments", 
Biometrika,  33,  305-325  and  328-332,  1946. 

[20]  Saaty,  T.  L.,  The  Analytic  Hierarchy  Process.,  McGraw-Hill  Book  Company,  New 
York  1980. 

[2 1  ]  Shedlowski,  Daniel  J.,  Army  Dollar  Resource  Allocation  II  Study,  CAA-SR-78-3,  US 
Army  Concepts  Analysis  Agency,  Bethesda  MD,  1978. 


2  4 


[22]  U.S.  Army  Cost  and  Economic  Analysis  Center  (USACEAC)  Instruction  DCA-P- 
92(R),  May  1991. 

[23]  U.S.  Army  Regulation  1-1,  Planning,  Programming,  Budgeting,  and  Execution  System, 
Washington  D.C.,  February  1990. 

[24]  U.S.  Army  Training  and  Doctrine  Command  (TRADOC),  CORBAN  (Corps  Battle 
Analyzer)  Users'  Guide,  July  1990. 

[25]  Yoon,  K.,  Systems  Selection  by  Multiple  Attribute  Decision  Making,  Ph.D.  Dissertation, 

Kansas  State  University,  Manhattan,  KS,  1980. 


25 


